Textual Inference for Retrieving Labeled Object Descriptions

This thesis presents a knowledge-based solution for retrieving English descriptions for objects in a collection. Based on detailed analysis of the errors made by a baseline system relying on surface-level features (i.e. term frequency), we infer that an ideal solution to this problem should use deeper representations of the meaning encoded in textual descriptions. Applied Textual Inference (ATI) as used in this thesis refers to the class of generic task-based evaluations that address this need. ATI tasks are challenge problems. Because they are intended to drive research on text understanding, the problems are designed to be hard enough to require reasoning. However in order to support cross-site comparisons of results, the problems are evaluated at the surface level. Examples include recognizing textual entailment (RTE), paraphrasing, summarization, word-replacement, and some types of question answering (QA). Deep representations and knowledge-based techniques have come to play an important role in state-of-the-art solutions for applied textual inference. However, to adapt them effectively for a particular ATI task, we must first understand that task and the details of why more shallow techniques fail. This thesis frames the problem of image description retrieval as an instance of ATI, and demonstrates how an inference engine and a set of symbolic knowledge resources in the form of ontologies can improve performance on this task, as measured by Mean Reciprocal Rank. In the process, we describe the results of several sub-tasks, each of which represents a contribution to the understanding of this problem and to the discovery and implementation of a knowledge-based solution: Introduce an image retrieval task supported by a data set containing over 50,000 images, hand-labeled with multiple descriptions; present a series of parameterizations for calculating the similarity between two descriptions; identify classes of